New technological developments allow the creation of more and more immersive multimedia systems. Three dimensional (3D) images and spatialized sound are now present in the end-user living space. But these systems are still limited to the stimulation of two senses, sight and hearing, while research in virtual reality has shown that haptic perception seems to be strongly connected to an immersive feeling.
From this research effort, a new scientific field called haptic-audiovisual (HAV) has emerged. HAV, introduced by A. E. Saddik in 2011, relates to the combination of haptics and audiovisual content. Since this is a recent area, all mechanisms, tools and concepts have yet to be specified (see FIG. 1).
The MPEG-V (Moving Picture Experts Group—Media Context and Control) architecture is one formalization of this workflow for producing, distributing, and rendering HAV content. The MPEG-V standard describes “Sensory Effects” which includes haptic effects but also gustatory and olfactory effects.
MPEG-V supports several haptic effects such as vibrations, rigid body motion or force-feedback. The effects can also stimulate the user's entire body or target a specific point in a space shared by the user (see example in FIG. 2).
MPEG-V targets the stimulation of the entire user's body and also targets the stimulation of a specific point in the space in which the user moves. In a same way, the haptic devices, also called actuators, can be defined by setting their capabilities, in term of ability to render haptic effects, and their location regarding the user's space. According to MPEG-V, only a point in the user's space is defined to render the haptic effects which may be an issue when the user moves in the space. In such a case, the haptic effect will not be felt by the user.
However, this description of the location of the effects does not allow a user to directly address a part of the user's body. Only a point in a space defined by a cube is given, for example, centerleft:middle:front in FIG. 2.
This representation does not take into account the human haptics perception. For example, it is known that if two vibrating stimuli are too close on the skin, only one vibration will be felt. From such a description of haptic effects, useless information may be stored or transmitted to a haptic renderer.
Moreover, a single update rate is specified. However, several locations on the user's body which are stimulated can have different temporal sensitivities.
The MPEG-V standard proposes a high-level description of haptic effects. However, the standard does not take into account human perceptual specificities and thus can encapsulate redundant data with a too low or a global spatial and temporal granularity.
In EP Application 14305823.8, the use of a detailed 3D body model, a skeleton, for example, was introduced to relate the haptic effects and the actuator inputs. Here the body model was used to infer the actuator inputs from an effect which could happen from a point not located at the actuator place through the use of physical model.